Sparse light field representation

ABSTRACT

The disclosure provides an approach for generating a sparse representation of a light field. In one configuration, a sparse representation application receives a light field constructed from multiple images, and samples and stores a set of line segments originating at various locations in epipolar-plane images (EPI), until the EPIs are entirely represented and redundancy is eliminated to the extent possible. In addition, the sparse representation application determines and stores difference EPIs that account for variations in the light field. Taken together, the line segments and the difference EPIs compactly store all relevant information that is necessary to reconstruct the full 3D light field and extract an arbitrary input image with a corresponding depth map, or a full 3D point cloud, among other things. This concept also generalizes to higher dimensions. In a 4D light field, for example, the principles of eliminating redundancy and storing a difference volume remain valid.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. provisional application havingSer. No. 61/820,104, filed on May 6, 2013, which is hereby incorporatedby reference in its entirety.

BACKGROUND

1. Field

This disclosure provides techniques for representing light fields and,in particular, a sparse representation of a light field.

2. Description of the Related Art

Light fields are typically constructed from a large set of images of ascene, captured at different viewing positions. One can capture a lightfield using a single photo diode, an array of cameras, a single camerataking a single image with a coded aperture, etc. At its simplest, thelight field may be captured by simply taking a number of photographs.

Light fields having high spatial-angular resolution, such as 3D lightfields constructed from hundreds of high resolution 2D images with theirrespective optical centers distributed along a 1D line or light fieldscreated with a camera array, can take up enormous storage space. Inaddition, such light fights may be difficult to process efficiently if,for example, the full input light field must be kept in memory.

SUMMARY

One aspect of the disclosure includes a computer implemented method forgenerating a sparse representation of a light field. The method includesreceiving the light field captured as a plurality of images depicting ascene. The method further includes receiving depth estimates of pointsin the scene, and determining an error between a reconstruction from thedepth estimates and the received images. In addition, the methodincludes storing the depth estimates and the error.

Other aspects include a computer-readable medium that includesinstructions that enable a processing unit to implement one or moreaspects of the disclosed methods as well as a system configured toimplement one or more aspects of the disclosed methods.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

So that the manner in which the above recited aspects are attained andcan be understood in detail, a more particular description of aspects ofthe disclosure, briefly summarized above, may be had by reference to theappended drawings.

It is to be noted, however, that the appended drawings illustrate onlytypical aspects of this disclosure and are therefore not to beconsidered limiting of its scope, for the disclosure may admit to otherequally effective aspects.

FIG. 1 illustrates an approach for estimating depth from highspatio-angular resolution light fields, according to an aspect of thedisclosure.

FIG. 2 illustrates a sparse representation for storing light field data,according to an aspect of the disclosure.

FIG. 3 illustrates a method for storing light field data in a sparserepresentation format, according to an aspect of the disclosure.

FIG. 4 illustrates a method for estimating depth in a scene, accordingto an aspect of the disclosure.

FIG. 5 illustrates a system in which an aspect may be implemented.

DETAILED DESCRIPTION

Aspects disclosed herein provide techniques for storing light fielddata. Given a light field constructed from multiple images, depthestimates of scene points are stored together with an error between areconstruction from the depth estimate and the multiple images.

FIG. 1 illustrates an approach for estimating depth from highspatio-angular resolution light fields, according to an aspect of thedisclosure. Illustratively, the light field may be a three-dimensional(3D) light field constructed from multiple high-resolutiontwo-dimensional (2D) images with optical centers distributed along aone-dimensional (1D) line. Panel A shows two such 2D images 101, 102,which may be captured in any feasible manner, such as with a cameraarray, using a camera on a motorized linear stage, etc. As discussed ingreater detail below, techniques disclosed herein are also generalizableto higher dimensions (e.g., four-dimensional (4D) input light fields andunstructured light fields).

A 3D light field with radiance values captured in RGB color space may bedenoted as a map L:

³→

³. The radiance rε

³ of a light ray may then be given as r=L(u, v, s), where s describesthe 1D ray origin and (u, v) represents the 2D ray direction. Here, smay be interpreted as different camera positions and/or camerasdistributed along a 1D line, and (u, v) may be interpreted as pixelcoordinates in a corresponding image I_(s)(u, v). It will be assumedherein, for the sake of simplicity, that u, v, s are regularly anduniformly sampled, i.e., the optical centers are uniformly spaced andall captured images are rectified, so that epipolar lines of a scenepoint coincide with the same horizontal scanline in all images. One wayto achieve such regular and uniform sampling is by mounting a camera ona motorized linear stage, capturing images at a uniform spacing (e.g., 1cm between camera positions), and approximating a regularly sampled 3Dlight field by correcting the captured images for lens distortion andcompensating for mechanical inaccuracies of the motorized linear stage(by, e.g., estimating the camera poses, computing the least orthogonaldistance line from all camera centers as a baseline, and then rectifyingall images with respect to the baseline).

A given u-v slice of the light field L for a fixed s corresponds toinput image I_(s), while a u-s slice for a fixed v coordinatecorresponds to an “epipolar-plane image” (EPI), which is, intuitively, astack of the same row v taken from all input images. Panel B illustratesan exemplary EPI 110. As shown in panels A and B, lines 101 a, 102 arepresent both the respective s-parameters of the two input images aswell as the v-parameter in the input images from which the EPI 110 isconstructed. Note, in panel A the lines 101 a, 102 a each mark a vcoordinate, whereas in panel B the lines each mark an s coordinate.

An EPI will be denoted herein as E_(v):

²→

³, with radiance r=E_(v) (u, s) of a ray at a position (u, s). EPIs of3D light fields may exhibit high ray coherence and include redundantinformation. The term EPI-pixel (u, s) is used herein instead of theterm ray at (u, s) for disambiguation. Further, the subscript v will beomitted for notational simplicity, as techniques disclosed hereinconsider mostly individual EPIs where the parameter v is fixed.

When the ray space of L is sampled densely enough, each scene pointappears as a line segment in an EPI with the slope of the line segmentdepending on the scene point's depth. Panel B shows examples of suchline segments 111, having slopes that depend on depth. Let Γ be the setof line segments originating at various locations in the input EPI E.Constructing Γ amounts to computing line slopes at the EPI pixels, i.e.,estimating the depth of scene points. As discussed in greater detailbelow, a sparse representation application (not shown) may employ afine-to-coarse strategy which estimates depth at edges in an EPI at ahighest image resolution first. Estimated depths are then propagatedthroughout the EPI (in s-direction), after which the EPI is iterativelydownsampled to coarser resolutions, and depths which were not previouslydetermined are estimated in a similar manner at those coarserresolutions. Note, this depth estimation strategy is optional. Anyfeasible technique that permits computation of depth estimates may beused, such as other stereo reconstruction methods over multi-view stereoor active scanners (e.g., a Kinect, laser scanner, structured lightscanner). The sparse light field representation discussed herein isadvantageous in that, even if the depth estimates are wrong, computationof the “difference EPI” still permits the original input images to berepresented (the difference EPI simply gets less sparse and compressionless good, as it has to compensate for errors of a poor depthestimator).

The estimated depths may be used to produce depth maps, e.g., depth maps121, 122 illustrated in panel C, which have a number of applications.Aside from generating a 3D model of the scene, the depth maps 121, 122may be used in automatic segmentation and image-based rendering, amongother things. For example, in automatic segmentation such as backgroundremoval, pixels within a prescribed depth interval may simply bethresholded. As another example, the scene may be directly visualizedusing the depth maps as a colored 3D point cloud via splat-basedrendering, with the ability to look around occluding objects.

FIG. 2 illustrates a sparse representation for storing light field data,according to an aspect of the disclosure. As discussed, in denselysampled ray spaces, each scene point may appear as a line segment in anEPI, which is a u-s slice of the light field L for a fixed v coordinate.Panel A shows an exemplary EPI 210.

The slope of line segments in the EPI correspond to respective scenepoints' depths, and EPIs of 3D light fields typically exhibit highcoherence and contain redundant information that can be utilized for amore efficient representation. In particular, rather than storing thefull EPI, the EPI may be reconstructed by knowing the parameters of linesegments. The sparse light field representations disclosed hereinutilizes this fact, and further, specifically considers completeness andvariation of the represented light field. Completeness of arepresentation may be compromised where a large amount of captured lightrays are occluded in any part of the EPI. Variation in the light fieldmay arise where scene points change their color along theircorresponding line segment in the EPI due to specularities or otherview-dependent effects. As a result of such occlusions and light fieldvariations, simply collecting and storing line segments and their coloralong single horizontal lines of the EPI may lack completeness and maynot capture the variations in the light field.

Aspects disclosed herein address the issues of occlusions and lightfield variations discussed above. According to one aspect, a sparserepresentation application samples and stores a set Γ of line segmentsoriginating at various locations in the input EPI E, until the entireEPI is represented and redundancy is eliminated to the extent possible,as discussed in greater detail below. In addition, the sparserepresentation application may determine and store a difference EPI ΔEthat accounts for variations in the light field. More specifically, theslope m of a line segment associated with a scene point at distance z isgiven by

$\begin{matrix}{{m = {\frac{1}{d} = \frac{z}{f_{x}b}}},} & (1)\end{matrix}$where d is the image space disparity defined for a pair of imagescaptured at adjacent positions or, equivalently, the displacementbetween two adjacent horizontal lines in the EPI, f_(x) is thehorizontal focal length in pixels and b is the metric distance betweeneach adjacent pair of imaging positions. Correspondingly, an EPI linesegment may be compactly described by a tuple l=(m, u, s, r^(T)), wherer is the average color of the scene point in the EPI. Γ is then the setof all tuples l. Techniques for generating the tuples l are discussed ingreater detail below. Additional tuples may be stored in regions whichare occluded, thereby ensuring completeness. From the set Γ of linesegments, a reconstructed EPI Ê may be generated by rendering the linesegments in the order of decreasing slopes, i.e., rendering the scenepoints from back to front. In one configuration, Γ may be stored as anordered list of tuples ln order of decreasing slopes for efficient EPIreconstruction.

In addition to Γ, the sparse representation application may determine adifference ΔE=E−Ê between the input EPI E and the reconstruction Ê.Note, the difference ΔE, also referred to herein as the “error,”captures the variation and detail information in the light field, suchas view dependent effects. Panel B illustrates an exemplary error EPI220. A high value of ΔE may occur, e.g., at specularities and atinaccurate slope estimates.

The Γ and ΔE for all EPIs generated from the input images, takentogether, compactly store all relevant information that is necessary toreconstruct the full 3D light field as well as extract an arbitraryinput image with a corresponding depth map, or a full 3D point cloud,among other things. One such Γ and ΔE 220 pair is shown in panel B.Experience has shown that this sparse representation may be stored using5-20% of the space required to store some red-green-blue (RGB) EPIs.

FIG. 3 illustrates a method 300 for storing light field data in a sparserepresentation format, according to an aspect of the disclosure. Asshown, at step 310, a sparse representation application receives aplurality of images of a scene, which together form a light field. Inone configuration, a 3D light field may be constructed from multiplehigh-resolution 2D images with optical centers distributed along a 1Dline, as discussed above.

At step 320, the sparse representation application receives depthestimates of scene points. Such depth estimates may be made by thesparse representation application itself using the received images(i.e., the depth estimates received are also made by the sparserepresentation application). However, the depth estimates need not bemade by the sparse representation application. For example, the depthestimates may simply be retrieved from a storage location.

In one configuration, estimating depth of scene points may be equivalentto determining the line slopes at EPI pixels. Such line slopes may beused to construct the set Γ of tuples l=(m, u, s, rT), which may be anordered list of tuples ln order of decreasing slopes for efficient EPIreconstruction.

In one configuration, the sparse representation application may employ afine-to-coarse estimation approach to estimate depth. In such a case,regions where the depth estimation is expected to perform well may firstbe identified in the full-resolution of an EPI E using an edgeconfidence test. In one configuration, the sparse representationapplication may compute edge confidence measures C_(e) for eachEPI-pixel and generate a binary mask M_(e) having value 1 for pixels forwhich C_(e) is greater than a given threshold, and 0 otherwise.

Using binary mask M_(e), the sparse representation application maygenerate depth estimates for EPI-pixels with high edge confidence. Asdiscussed in greater detail below, the sparse representation applicationmay, in one configuration, test a number of discrete depth hypotheses dand pick a depth that leads to a highest color density of sampledEPI-pixels. The depth estimate itself is used to improve the initialconfidence toward a refined depth confidence C_(d), which provides agood indicator of the reliability of particular depth estimates. Thedepth estimate may also be propagated throughout the EPI. This processof depth estimation and propagation may be iterated until all EPI-pixelswith high edge confidence have been processed. After one iteration,sufficiently detailed regions at the highest resolution level of the EPIE have a reliable depth value assigned, while the depth in morehomogenous regions (which were masked out) remain unknown.

After estimating depth at the full resolution, the sparse representationapplication may downsample the EPI E to coarser resolutions. At eachresolution level, the sparse representation application may determineedge confidence for EPI-pixels which are not yet processed, estimatedepths of EPI-pixels with high edge confidence, and propagate the depthestimates. This iterative procedure produces depth estimates for allEPI-pixels, and the procedure may be repeated for all EPIs.

At step 330, the sparse representation application determines an errorbetween a reconstruction from the depth estimates and the input imageswhich accounts for variations in the light field. As discussed, thesparse representation application may estimate (or receive) depth of ascene point as the slope m of a line segment associated with the pointat distance z using equation (1), and store each EPI line segmentcompactly as a tuple l=(m, u, s, r^(T)), where r is the average color ofthe scene point in the EPI. Given the set Γ of all tuples l, the sparserepresentation application may compute the error between a reconstructedEPI, having lines defined by the tuples l, and the EPI generated fromthe input images as the difference ΔE=E−Ê, where Ê is the reconstructionusing the set Γ of all tuples l. As discussed, a high value of ΔE mayoccur, e.g., at specularities and at inaccurate slope estimates.

At step 340, the sparse representation application stores the depthestimates and the error in a storage device. In one configuration, thesparse representation application may store the set Γ of all tuples l,as well as the difference EPI ΔE, discussed above. Taken together, the Γand ΔE, for all EPIs generated from the input images, compactly storeinformation needed to reconstruct the full 3D light field, as well as toextract an arbitrary input image with a corresponding depth map, or afull 3D point cloud, among other things. Experience has shown that Γ andΔE may be stored using 5-20% of the space required to store somered-green-blue (RGB) EPIs. The stored Γ and ΔE may be used toreconstruct the full 3D light field, as well as extract an arbitraryinput image with a corresponding depth map, or a full 3D point cloud,among other things.

FIG. 4 illustrates a method 400 for estimating depth in a scene,according to an aspect of the disclosure. As discussed, in oneconfiguration, a sparse representation application may itself estimatedepth from received images. Illustratively, the method 400 begins atstep 410, where the sparse representation application receives aplurality of images of a scene, which together form a light field. Forexample, a 3D light field may be constructed, e.g., from multiplehigh-resolution 2D images with optical centers distributed along a 1Dline, as discussed above.

At step 420, the sparse representation application generates EPIs fromthe plurality of images. For example, for 3D light fields, an EPI may begenerated by simply stacking the same row v from all input images as au-s slice of the light field L for a fixed v coordinate corresponds toan EPI.

For a given EPI, the sparse representation application determines edgeconfidence at step 430. In one configuration, the sparse representationapplication may compute the following difference measure:C _(e)(u,s)=Σ_(u′εN(u,s)) ∥E(u,s)−E(u′,s)∥²,  (2)where N(u, s) is a 1D window in EPI E around pixel (u, s) and may be asmall neighborhood (e.g., 9 pixels), as it may simply measure localcolor variation. C_(e) may be fast to compute, and may be thresholded togive a binary confidence mask M_(e). For example, the threshold may havevalue 0.02, with the binary confidence mask M_(e) having value 1 forpixels for which C_(e) is greater than 0.02, and 0 otherwise. Spuriousisolated regions may also be removed by, e.g., applying a morphologicalopening operator to the mask M_(e).

At step 440, the sparse representation application computes depthestimates for pixels in the given EPI which are above an edge confidencethreshold. Such EPI-pixels with high reliability are stored as tuples ln1 in the set Γ of tuples and, as discussed in greater detail below,propagated throughout the EPI. Where a binary confidence mask M_(e) isdetermined, the sparse representation application may compute depthestimates for those EPI pixels marked as confident in M_(e) to preventcomputing depth estimates at ambiguous EPI pixels, thereby speeding updepth computation without sacrificing accuracy. In one configuration,the depth estimates may be computed per scanline of an EPI, i.e., afixed parameters may be selected and a depth estimate computed for E(u,ŝ) with M_(e)(u, ŝ)=1. In one configuration, the sparse representationapplication may begin by settings ŝ to a scanline at the center of theEPI, estimate depth for EPI-pixel of that scanline, propagate thoseestimates as discussed in greater detail below, then set ŝ to thenearest s with respect to the center of the EPI that still hasunprocessed pixels, and so forth, until all edge-confident EPI-pixelshave been processed or masked out during propagation.

The sparse representation application may assign a depth z, orequivalently a disparity d, to each EPI-pixel (u, ŝ). For a hypotheticaldisparity d, the set R of radiances or colors of EPI-pixels may besampled asR(u,d)={E(u+(ŝ−s)d,s)|s=1, . . . , n},  (3)where n is the number of views in the light field. From the density ofradiance values in R(u, d), a depth score S(u, d) may be computed inlinearized RGB color space. This assumes that that the scene isessentially Lambertian, i.e., that a set is likely to represent anactual scene point if the radiance samples are densely positioned in theunderlying color space.

In one configuration, the sparse representation application may computethe density using iterations of a modified Parzen window estimation withan Epanechnikov kernel, with the initial depth score defined as

$\begin{matrix}{{{S\left( {u,d} \right)} = {\frac{1}{{\mathcal{R}\left( {u,d} \right)}}{\sum\limits_{r \in {\mathcal{R}{({u,s})}}}^{\;}\;{K\left( {r - \overset{\_}{r}} \right)}}}},} & (4)\end{matrix}$where r=E(u, ŝ) is the radiance value at the current EPI-pixel, and thekernel is

${K(x)} = {{1 - {{\frac{x}{h}}^{2}\mspace{14mu}{if}\mspace{14mu}{\frac{x}{h}}}} \leq 1}$and 0 otherwise. Here, h is the bandwidth parameter and may be set to,e.g., h=0.02. In alternative configurations, other kernels may be usedsuch as Gaussian or other bell-shaped kernels. To reduce the influenceof noisy radiance measurements, the sparse representation applicationmay compute an iteratively updated radiance mean

$\left. \overset{\_}{r}\leftarrow\frac{\sum\limits_{r \in \mathcal{R}}^{\;}\;{{K\left( {r - \overset{\_}{r}} \right)}r}}{\sum\limits_{r \in \mathcal{R}}^{\;}\;{K\left( {r - \overset{\_}{r}} \right)}} \right.$to plug into equation (4). Experience has shown that robustness to noisemay be achieved after only a few iterations, e.g., 10 iterations.

For each EPI pixel (u, ŝ), the sparse representation application maycompute scores S(u, ŝ) for an entire range of admissible disparities d,and assign the disparity with the highest score as the pixel's depthestimateD(u,ŝ)=arg max_(d) S(u,d).  (5)

The sparse representation application may further compute a refinedconfidence C_(d) as a measure of reliability of the depth estimate.Low-confidence depth estimates may be discarded and marked forre-computation at a later stage. In one configuration, the refinedconfidence C_(d) may combine the edge confidence C_(e) with thedifference between the maximum score S_(max)=max_(d) S(u, d) and theaverage score S=Σ_(d)S(u, d)C _(d)(u,ŝ)=C _(e)(u,ŝ)∥S _(max) −S∥.  (6)Note, the refined confidence C_(d) combines two complementary measures.For example, noisy regions of an EPI would result in a highedge-confidence C_(e), while a clear maximum S_(max) is not available.Similarly, ambiguous homogenous regions in an EPI, where C_(e) is low,can produce a strong, but insufficiently unique S_(max). Each confidentdepth estimate, i.e., D(u, ŝ) where C_(d)(u, s)>ε, may be stored as aline segment tuple l=(m, u, s, r ^(T)) in Γ, where r represents the meanradiance of R(u, d), and m is the slope of the line segment computedfrom d as m=1/d.

In one configuration, the sparse representation application may apply amedian filter on the computed depths to attempt to eliminate theinfluence of outliers. A straightforward median filter may not result inthe precise localization of silhouettes. The sparse representationapplication may instead use a bilateral median filter that preserves thelocalization of depth discontinuities by leveraging information from theradiance estimates of nearby EPIs. Such a bilateral median filter may beimplemented by replacing depth values D_(v)(u, ŝ) with the median valueof the set{D _(v′)(u′,ŝ)|(u′,v′,ŝ)εN(u,v,ŝ),∥E _(v)(u,ŝ)−E _(v′)(u,ŝ)∥<εM_(e)(u′,v′,ŝ)=1},  (7)where (u′, v′, ŝ)εN(u, v, ŝ) denotes a small window over I_(ŝ) (e.g., awindow of size 11×11 pixels), and the color tolerance threshold ε is,e.g., ε=0.1. Note, increasing the color tolerance ε and the kernelbandwidth h, discussed above, compared to other default values increasesrobustness to noise, whereas small values of ε and h preserve finerdetails. In equation (7), the second condition ensures that onlyEPI-pixels of similar radiance are considered, and the third conditionmasks out EPI-pixels lacking confidence for which no depth estimation isavailable.

At step 450, the sparse representation application propagates the depthestimates to other EPI-pixels. As discussed, each confident depthestimate D(u, ŝ) may be stored as a line segment tuple l=(m, u, s, r^(T)) in Γ, where r is the mean radiance of (u, ŝ), namely

$\left. \overset{\_}{r}\leftarrow{\frac{\sum\limits_{r \in \mathcal{R}}^{\;}\;{{K\left( {r - \overset{\_}{r}} \right)}r}}{\sum\limits_{r \in \mathcal{R}}^{\;}\;{K\left( {r - \overset{\_}{r}} \right)}}.} \right.$In one configuration, the depth estimate may be propagated along theslope of its corresponding EPI line segment to all EPI-pixels (u′, ŝ)having a radiance similar to the mean radiance, i.e., ∥E(u′, s′)−r∥<ε.This is a conservative visibility estimate and ensures that foregroundobjects in the EPI are not overwritten by background objects duringpropagation. In an alternative configuration, the sparse representationapplication may use full mean shift clustering, and propagate the depthestimate to cluster elements. As discussed, the sparse representationapplication may, after depth estimate propagation, set ŝ to the nearests with respect to the center of the EPI that still has unprocessedpixels, and so forth, until all edge-confident EPI-pixels have beenprocesses or masked out by propagation.

At step 460, the sparse representation application iterativelydownsamples the EPI and repeats, after each down-sampling, thedetermining of edge confidence of step 430, computing of depth estimatesof step 440, and propagating of depth estimates of step 450. Parts ofthe EPI without assigned depth values were either previously ambiguousdue to homogenous colors (i.e., insufficient edge confidence), or havestrongly view-dependent appearance (i.e., insufficient depthconfidence). A fine-to-coarse strategy may be used to compute depth insuch less detailed and less reliable regions by exploiting theregularizing effect of an iterative downsampling of the EPI. Further, toenhance robustness and efficiency, the sparse representation applicationmay use previously computed confident depth estimates as depth intervalbounds for depth estimation at coarser resolutions.

In one configuration, the sparse representation application may use theupper and lower bounds of the closest reliable depth estimates in eachhorizontal row of the EPI as the depth bound estimates. Then the sparserepresentation application may downsample the EPI (e.g., by a factor of0.5) along the spatial u and v directions, while the resolution alongthe angular s-dimension is preserved. The sparse representationapplication may smooth the EPI along the spatial dimensions using, e.g.,a 7×7 Guassian filter with standard deviation σ=√{square root over(0.5)} to avoid aliasing. After downsampling, the sparse representationapplication may perform edge confidence estimation, depth estimation,and propagation at the coarser resolution. EPI-pixels with reliabledepth estimates computed at higher resolutions may not be consideredagain, aside from being used for deriving the depth bounds discussedabove. This fine-to-coarse process may be iterated through all levels ofthe EPI pyramid until any of the image dimensions becomes less than agiven size (e.g., 10 pixels). At the coarsest level, the sparserepresentation application may assign depth estimates to all pixelsregardless of confidence.

At step 470, the sparse representation application successivelyupsamples depth estimates at coarse resolution levels to the respectivehigher resolution levels and assigned to the correspondinghigher-resolution EPI-pixels without a depth estimate, until allEPI-pixels at the finest resolution level have a corresponding depthestimate. In one configuration, the sparse representation applicationmay remove spurious speckles by, e.g., applying a 3×3 median.

If there are more EPIs, then at step 480, the method 400 returns to step430, and another EPI is processed.

FIG. 5 depicts a block diagram of a system in which an aspect may beimplemented. As shown, the system 500 includes, without limitation, acentral processing unit (CPU) 510, a network interface 530, aninterconnect 515, a memory 560 and storage 520. The system 500 may alsoinclude an I/O device interface 540 connecting I/O devices 550 (e.g.,keyboard, display and mouse devices) to the system 500.

The CPU 510 retrieves and executes programming instructions stored inthe memory 560. Similarly, the CPU 510 stores and retrieves applicationdata residing in the memory 560. The interconnect 515 facilitatestransmission, such as of programming instructions and application data,between the CPU 510, I/O device interface 540, storage 520, networkinterface 530, and memory 560. CPU 510 is included to be representativeof a single CPU, multiple CPUs, a single CPU having multiple processingcores, and the like. In other aspects, one or more graphics processingunits (GPUs) may be used in lieu of, or in conjunction with, the CPU510. And the memory 560 is generally included to be representative of arandom access memory. The storage 520 may be, e.g., a disk drive storagedevice. Further, system 500 is included to be representative of aphysical computing system as well as virtual machine instances hosted ona set of underlying physical computing systems. Further still, althoughshown as a single computing system, one of ordinary skill in the artwill recognized that the components of the system 500 shown in FIG. 5may be distributed across multiple computing systems connected by a datacommunications network.

As shown, the memory 560 includes an operating system 561 and sparserepresentation application 562. Illustratively, the operating system mayinclude Microsoft's Windows®. The sparse representation application 562is configured to determine depth estimates based on light fields andstore a sparse light field representation which includes the depthestimates and an error between a reconstruction from the estimates andinput light field images. In one configuration, the sparserepresentation application 562 may store a set Γ of tuples l=(m, u, s,r^(T)) generated from received depth estimates, determine a differenceΔE=E−Ê between the input EPI E and the reconstruction Ê generated byrendering the line segments described by the tuples l, and store the setΓ and the difference ΔE as a sparse representation 522 of the lightfield, according to the method 300 discussed above. In addition, thesparse representation application 562 may itself estimate depth based ongiven input images 521 by, generating EPIs, determining edge confidencefor EPI-pixels, estimating depths of EPI-pixels with high edgeconfidence, propagating the depth estimates, and repeating this processwith iteratively downsampled EPIs, according to the method 400 discussedabove.

Although discussed above primarily with respect to a loss-less sparselight field representation from which the full light field may bereconstructed, alternative configurations may also apply, e.g., lossycompression to further reduce the size of the representation. An exampleof such lossy compression is JPEG compression, which may be applied tothe difference EPI ΔE.

Although discussed above primarily with respect to a regularly sampled3D light field, techniques disclosed herein may be readily adapted toother light fields, such as 4D light fields and unstructured lightfields. In a regular 4D light field, camera centers are horizontally andvertically displaced, leading to a 4D parametrization of rays as r=L(u,v, s, t), where t denotes the vertical ray origin. The ray sampling ofequation (3) may then be extended toR(u,v,s,t,d)={L(u+(ŝ−s)d,v+({circumflex over (t)}−t),d,s,t)|s=1, . . . ,n,t=1, . . . , m},  (8)where (ŝ, {circumflex over (t)}) is the considered view and m denotesthe number of vertical viewing positions. As a result, sampling may beperformed in a 2D plane in a 4D ray space, also referred to herein as a“3D epipolar (plane) volume,” as opposed to the 1D line in the case of3D light fields. The depth propagation may take place along both s- andt-directions. Additionally, the tuples l may represent planes, and bestored together with reconstruction error ΔE, which may be a 3D errorepipolar (plane) volume.

For arbitrary, unstructured input (e.g., an unstructured light field orunstructured set of images), camera poses may be estimated to determinethe set of sampled rays for a depth hypothesis. For example, cameraposes may be estimated, using structure-from-motion techniques, from theinput images of a scene that are used to construct a light field. Havingthe camera poses means that the relative positions and orientations areknown of the camera viewpoints for all input images. With thisinformation available, it can be determined where a scene point isprojected in each input image, and the pixels collected from thoseprojected positions in input images form the set of sampled raysmentioned above.

In one configuration, each considered pixel may be sent to 3D space inaccordance with the hypothesized depth, and then the 3D position may bere-projected to the image coordinate systems of all other views toobtain the sampling positions. The set of sampled rays becomesR(u,v,s,d)={L(u′,v′,s)|s=1, . . . , n,P _(s) ⁻¹ [u′v′fd] ^(T) =P _(ŝ) ⁻¹[uvfd] ^(T)},  (9)where P_(s) denotes the camera projection matrix of view s, and f is thecamera focal length. The depth estimate may then be depth or disparitymaps or any other form of 3D representation/model of the depicted scene,such as a point cloud, triangle mesh, or any other surfacerepresentation. Such depth estimates may be stored together withreconstruction error ΔE.

Advantageously, techniques disclosed herein permit light fields havinghigh spatial-angular resolution to be stored in a compact representationthat reduces storage space requirements. The compact representation mayhandle both occlusions and light field variations, storing all relevantinformation necessary to reconstruct the full light field as well asextract an arbitrary input image with a corresponding depth map, or afull 3D point cloud, among other things. Further, the compactrepresentation enables efficient parallel processing of input lightfields, as the full light fields need not be kept in memory.

The foregoing description references aspects of the disclosure. However,it should be understood that the disclosure is not limited to specificdescribed aspects. Instead, any combination of the following featuresand elements, whether related to different aspects or not, iscontemplated to implement and practice the disclosure. Furthermore,although aspects of the disclosure may achieve advantages over otherpossible solutions and over the prior art, whether or not a particularadvantage is achieved by a given aspect is not limiting of thedisclosure. Thus, the following aspects, features, and advantages aremerely illustrative and are not considered elements or limitations ofthe appended claims except where explicitly recited in a claim(s).Likewise, reference to “the disclosure” shall not be construed as ageneralization of any inventive subject matter disclosed herein andshall not be considered to be an element or limitation of the appendedclaims except where explicitly recited in a claim(s).

Aspects of the present disclosure may be embodied as a system, method orcomputer program product. Accordingly, aspects of the present disclosuremay take the form of an entirely hardware aspect, an entirely softwareaspect (including firmware, resident software, micro-code, etc.) or anaspect combining software and hardware aspects that may all generally bereferred to herein as a “circuit,” “module” or “system.” Furthermore,aspects of the present disclosure may take the form of a computerprogram product embodied in one or more computer readable medium(s)having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus or device.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality and operation of possible implementations ofsystems, methods and computer program products according to variousaspects of the present disclosure. In this regard, each block in theflowchart or block diagrams may represent a module, segment or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. Each block of the block diagrams and flowchartillustrations, and combinations of blocks in the block diagrams andflowchart illustrations can be implemented by special-purposehardware-based systems that perform the specified functions or acts, orcombinations of special purpose hardware and computer instructions.

While the foregoing is directed to aspects of the present disclosure,other and further aspects may be devised without departing from thebasic scope thereof, and the scope thereof is determined by the claimsthat follow.

What is claimed is:
 1. A computer-implemented method for generating andstoring a compact representation of a light field, comprising: receivingthe light field captured as a plurality of images depicting a scene,wherein the light field is one of a three-dimensional (3D) light fieldand a four-dimensional (4D) light field; receiving depth estimates ofpoints in the scene; determining an error between a reconstruction fromthe depth estimates and the received images; and storing, in computerstorage hardware, the compact representation of the light field whichincludes the depth estimates and the determined error, wherein, when thelight field is the 3D light field, the depth estimates include slopes oflines at pixels of epipolar-plane images (EPIs) generated from theplurality of images, the reconstruction includes one or more EPIsgenerated based on the slopes, and the error is an error between theEPIs generated from the plurality of images and the one or morereconstructed EPIs, and wherein, when the light field is the 4D lightfield, the depth estimates include planes passing through pixels of 3Depipolar (plane) volumes generated from the plurality of images, thereconstruction includes one or more 3D epipolar (plane) volumesgenerated based on the planes, and the error is an error between the 3Depipolar (plane) volumes generated from the plurality of images and theone or more reconstructed 3D epipolar (plane) volumes.
 2. The method ofclaim 1, wherein, when the light field is the 3D light field, the depthestimates are stored as tuples each representing a respective one of thelines and having form l=(m, u, s, r^(t)), where m is a slope, (u, s) isa point, and r is an average color of the point (u, s) in one of theEPIs.
 3. The method of claim 2, wherein the tuples are stored in orderof decreasing slope.
 4. The method of claim 1, wherein, when the lightfield is the 4D light field, the depth estimates are stored as tuples leach representing a respective one of the planes.
 5. The method of claim1, wherein the light field further includes an unstructured light fieldor unstructured set of images, and wherein the depth estimates furtherinclude one of a depth map, a disparity map, and a 3D representation ormodel of the scene associated with the unstructured light field.
 6. Themethod of claim 1, wherein, when the light field is the 3D light field,the received depth estimates of points in the scene are estimated by:generating the EPIs from the plurality of images; iterativelydownsampling the EPIs to coarser resolutions; and at each of theiterations, for each of the EPIs: determining edge confidence scores forpixels of the EPI, determining depth estimates for the pixels of the EPIassociated with edge confidence scores above a threshold value, andpropagating the depth estimates to other pixels of the EPI.
 7. Themethod of claim 1, further comprising, reconstructing the light fieldbased on the stored depth estimates and the stored error.
 8. The methodof claim 1, further comprising, extracting an image and a correspondingdepth map, or a full point cloud, based on the stored depth estimatesand the stored error.
 9. A non-transitory computer-readable storagemedium storing instructions, which when executed by a computer system,perform operations for generating and storing a compact representationof a light field, the operations comprising: receiving the light fieldcaptured as a plurality of images depicting a scene, wherein the lightfield is one of a three-dimensional (3D) light field and a four-dimensional (4D) light field; receiving depth estimates of points inthe scene; determining an error between a reconstruction from the depthestimates and the received images; and storing, in computer storagehardware, the compact representation of the light field which includesthe depth estimates and the determined error, wherein, when the lightfield is the 3D light field, the depth estimates include slopes of linesat pixels of epipolar-plane images (EPIs) generated from the pluralityof images, the reconstruction includes one or more EPIs generated basedon the slopes, and the error is an error between the EPIs generated fromthe plurality of images and the one or more reconstructed EPIs, andwherein, when the light field is the 4D light field, the depth estimatesinclude planes passing through pixels of 3D epipolar (plane) volumesgenerated from the plurality of images, the reconstruction includes oneor more 3D epipolar (plane) volumes generated based on the planes, andthe error is an error between the 3D epipolar (plane) volumes generatedfrom the plurality of images and the one or more reconstructed 3Depipolar (plane) volumes.
 10. The computer-readable storage medium ofclaim 9, wherein, when the light field is the 3D light field, the depthestimates are stored as tuples each representing a respective one of thelines and having form l=(m, u, s, r^(T)), where m is a slope, (u, s) isa point, and r is an average color of the point (u, s) in one of theEPIs.
 11. The computer-readable storage medium of claim 10, wherein thetuples are stored in order of decreasing slope.
 12. Thecomputer-readable storage medium of claim 9, wherein, when the lightfield is the 4D light field, the depth estimates are stored as tuples leach representing a respective one of the planes.
 13. Thecomputer-readable storage medium of claim 9, wherein the light fieldfurther includes an unstructured light field or unstructured set ofimages, and wherein the depth estimates further include one of a depthmap, a disparity map, and a 3D representation or model of the sceneassociated with the unstructured light field.
 14. The computer-readablestorage medium of claim 9, wherein, when the light field is the 3D lightfield, the received depth estimates of points in the scene are estimatedby: generating the EPIs from the plurality of images; iterativelydownsampling the EPIs to coarser resolutions; and at each of theiterations, for each of the EPIs: determining edge confidence scores forpixels of the EPI, determining depth estimates for the pixels of the EPIassociated with edge confidence scores above a threshold value, andpropagating the depth estimates to other pixels of the EPI.
 15. Thecomputer-readable storage medium of claim 9, further comprising one ofreconstructing the light field based on the stored depth estimates andthe stored error and extracting an image and a corresponding depth map,or a full point cloud, based on the stored depth estimates and thestored error.
 16. A system, comprising: a processor; and a memory,wherein the memory includes an application program configured to performoperations for generating and storing a compact representation of alight field, the operations comprising: receiving the light fieldcaptured as a plurality of images depicting a scene, wherein the lightfield is one of a three-dimensional (3D) light field and afour-dimensional (4D) light field, receiving depth estimates of pointsin the scene, determining an error between a reconstruction from thedepth estimates and the received images, and storing, in computerstorage hardware, the compact representation of the light field whichincludes the depth estimates and the determined error, wherein, when thelight field is the 3D light field, the depth estimates include slopes oflines at pixels of epipolar-plane images (EPIs) generated from theplurality of images, the reconstruction includes one or more EPIsgenerated based on the slopes, and the error is an error between theEPIs generated from the plurality of images and the one or morereconstructed EPIs, and wherein, when the light field is the 4D lightfield, the depth estimates include planes passing through pixels of 3Depipolar (plane) volumes generated from the plurality of images, thereconstruction includes one or more 3D epipolar (plane) volumesgenerated based on the planes, and the error is an error between the 3Depipolar (plane) volumes generated from the plurality of images and theone or more reconstructed 3D epipolar (plane) volumes.